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rhe  purpose  of  this  article  is  to  document  lessons  learned  from  the  Joint 
STARS  experience ,  written  primarily  from  a  program  office  perspective , 
and  provide  suggestions  for  future  programs  facing  DAB  reviews.  In 
May  1993 ',  the  Joint  STARS  (the  Airborne  Standoff  Target  Acquisition  Recog¬ 
nition  System  used  during  Desert  Storm)  program  underwent  a  Defense  Acqui¬ 
sition  Board  (DAB)  review  for  Low  Rate  Initial  Production  (LRIP).  Originally 
established  in  the  program  baseline  as  DAB  IIIA ,  this  event  was  recast  as  the 
“Program  Review  for  LRIP,  ”  since  the  revised  DoD  Instruction  5000.2  no 
longer  provides  for  a  DAB  IIIA.  However,  the  program  review  otherwise 
complied  with  almost  all  DAB  procedures  and  documentation  requirements. 


A  GOOD  LANDING  STARTS  WITH  THE  APPROACH 

Success  —  coming  up  golden  —  in  the  DAB  process  depends  on  “consistency, 
consistency,  consistency !”  Although  each  program  follows  itsown  course,  DoD 
Instruction  5000.2  presumes  that  life  begins  with  a  Mission  Needs  Statement, 
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Figure  1.  From  DoDI  5000.2 


from  which  a  number  of  alternatives  were  explored,  including  changes  in  tactics, 
modifications  to  existing  systems,  and  development  of  new  systems  such  as  the 
subject  of  the  DAB.  This  model  further  presumes  your  particular  system  is  being 
recommended,  because  a  Cost  and  Operational  Effectiveness  Analysis  (COEA) 
so  indicated.  In  addition,  it  identified  critical  operational  characteristics  that 
flowed  into  the  Operational  Requirements  Document  (ORD),  Acquisition 
Program  Baseline  (APB),  Test  and  Evaluation  Master  Plan  (TEMP),  system 
specification,  and  all  other  aspects  of  your  program. 

While  few  if  any  programs  actually  evolve  this  way,  the  system  will  tend  to 
force  the  evaluation  of  your  program  in  terms  of  the  model,  and  to  the  extent  that 
a  program  conforms,  the  dialogue  is  likely  to  be  crisper.  If  your  program  happens 
to  resemble  the  model  program,  the  risks  are  manageable,  and  the  program  is 
affordable,  write  it  all  down  in  the  DAB  documentation  and  you  should  be 
golden.  If  it  differs  and  you  have,  say,  two  years  to  spruce  up  the  foundation,  the 
investment  is  well  worth  it. 

Since  the  process  involves  both  pol  itical  and  technical  considerations,  spend 
timebefore  the  DAB1  to  make  itpolitically  palatable.  This  includes  strategies  for 
cooperative  development,  interoperability,  consideration  of  acquisition  strate- 


1  “DAB”  in  Pentagon  jargon  means  both  the  process  and  the  meeting  (review)  itself. 
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gies  currently  in  vogue,  and  many  other  aspects. 

PROCESS  OVERVIEW 

The  DoDI  5000.2,  Part  13,  contains  core  guidance  for  DAB  reviews.  As  shown 
in  Figure  1,  the  process  formally  begins  with  a  planning  meeting.  Subsequently, 
draft  program  documentation  is  prepared  for  review  by  staff  of  the  Office  of  the 
Secretary  of  Defense  (OSD).  This  commences  a  series  of  further  reviews  to 
assess  documentation  adequacy,  some  requiring  OSD  approval,  and  identify 
issues  needing  resolution. 

Let  me  note  now  that  a  DAB  cannot  be  done  within  the  specified  180  days: 
(1)  the  time  is  not  sufficient  to  prepare  and  coordinate  the  documentation;  (2)  the 
draft  Cost  Analysis  Requirements  Description  (CARD)  must  be  submitted  at  the 
planning  meeting;  and;  (3)  several  processes,  notably  the  independent  cost 
estimate,  take  longer  than  six  months. 

DOCUMENTATION  REQUIREMENTS  OVERVIEW 
Except  for  the  CARD,  required  milestone  documentation  is  listed  in  DoDI 
5000.2,  Part  11,  Section  C,  and  guidance  for  preparation  is  provided  in  the 
accompanying  manual,  DoD  5000.2-M.  Preparation  entails  the  concerted  ef¬ 
forts  of  the  program  office,  the  using  command,  operational  test  agency, 
program  executive  officer  and  mission  area  director.2 

The  exercise  is  onerous,  but  all  involved  must  appreciate  that  a  favorable 
DAB  outcome  depends  on  affirmative  OSD  staff  recommendations  provided  in 
the  Integrated  Program  Assessment  (IP A)  (shown  in  Figure  2).  If  documenta¬ 
tion  detail  is  inadequate,  additional  actions  by  the  program  office,  or  delays, 
could  ensue. 

For  some  documents,  such  as  the  Integrated  Program  Summary  (IPS)  and 
TEMP,  the  directive  clearly  specifies  the  staffing  process.  For  others,  such  as  the 
Manpower  Estimate  Report  (MER),  unclear  instructions  make  it  sometimes 
difficult  to  determine  who  coordinates  on  the  document.  In  effect,  OSD  staff 
have  approval  authority  over  all  of  documentation,  and  can  recommend  post¬ 
poning  the  DAB  or  committee  review. 

Part  11  of  DoDI  5000.2  lists  documentation  needed  for  a  milestone  review. 
Other  parts  proscribe  life  cycle  documents.  They  include,  but  are  not  limited  to: 

•  Systems  Engineering  Management  Plan  (SEMP) 

•  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP) 


2  Under  the  current  Air  Force  acquisition  structure,  the  program  office  for  most  major 
programs  reports  to  the  Program  Executive  Officer  (PEO),  while  the  Program  Element 
Monitor  (PEM)  resides  in  a  Mission  Area  Directorate  on  the  SAF/AQ  staff. 
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•  Configuration  Management  Plan 

•  Training  Development  Plan 

•  Human  Systems  Integration  Plan  (HSIP) 

•  Program  Protection  Plan  (PPP) 


MILESTONE  DOCUMENTATION  CONCEPT 


Figure  2.  From  DoDI  5000.2 


Documentation  requirements  are  agreed  to  at  the  planning  meeting.  In  the 
Joint  STARS  DAB,  OSD  staffers  included  the  Program  Protection  Plan  and  the 
Human  Systems  Integration  Plan  they  needed  for  the  IPA.3  Staffers  also 
requested  a  Total  Quality  Management  Implementation  Plan,  unmentioned  in 
DoD  5000.2.  Though  the  program  office  should  be  wary  about  volunteering  for 
unnecessary  work  by  asking  if  a  particular  document  is  required,  avoiding  later 
surprises  is  important. 


3  As  a  result  of  previous  reviews  and  discussions  with  OSD  staff,  a  formal  Joint 
Requirements  Oversight  Council  (JROC)  review  was  not  held,  nor  was  a  COEA  update 
required. 


164  -  Spring  1994 


Acquisition  Review  Quarterly 


Coining  Up  Golden: 

Defense  Acquisition  Board  Review  Guide  for  Program  Offices 


DOCUMENT  INTERRELATIONSHIPS 


Figure  3.  Chart  from  DoDI  5000.2 


DOCUMENT  INTERRELATIONSHIPS 

Figure  3  shows  major  interrelationships  between  documents,  and  one  could 
argue  for  several  more  lines.  Many  are  direct  feeds,  parts  of  a  particular 
document  either  lifted  from  or  summarized  in  another,  requiring  consistency  and 
accuracy.  For  example,  the  program  office  must  summarize  the  life-cycle  cost 
estimate  in  the  IPS.  Similarly,  APB  performance  thresholds  and  objectives  are 
included  in  the  development  test  report  as  baselines  against  which  results  are 
reported. 

OBSERVATIONS  ON  SPECIFIC  DOCUMENTS 
CARD 

The  CARD  has  the  longest  lead  time,  is  required  at  the  planning  meeting  in  draft, 
and  drives  the  independent  cost  estimate,  which  is  another  long  lead  activity.  The 
CARD  contains  a  system  description,  a  Work  Breakdown  Structure  (WBS), 
subsystem  descriptions,  software  descriptions,  schedules,  test  program  descrip¬ 
tions  (consistent  with  the  TEMP)  logistics  concepts  (consistent  with  the  Inte¬ 
grated  Logistic  Support  Plan  (ILSP)),  and  many  other  detailed  aspects  of  the 
program.  While  it  provides  the  basis  for  a  cost  estimate,  it  does  not  document 
anything  in  dollars,  but  provides  requirements  which  estimators  price  out.  With 
its  detailed  technical  content,  it  may  be  the  most  suitable  document  for  prime 
contractor  input. 
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ORD 

The  ORD  is  the  cornerstone  for  the  entire  building.  It  provides  the  basis  for  the 
COEA,  program  content,  criteria  for  developmental  and  operational  testing,  and 
other  bits  and  pieces  of  program  documentation.  Its  performance  thresholds, 
objectives  and  technical  parameters  are  included  in  other  documents  such  as  the 
APB  and  TEMP.  Don’t  be  surprised  if  cost  analysts  or  logistics  staff  seek 
information  from  it. 

The  perfect  ORD  has  not  been  written.  The  ORD  developer  is  constantly  told 
to  write  requirements  in  operational  terms,  yet  the  less  specific  the  requirement 
is,  the  more  open  it  is  to  interpretation  by  the  tester  or  evasion  by  the  developer. 
It  must  be  straightforward  and  defendable  before  JROC  and  senior  military 
management  and  staff  review  —  particularly  the  operational  testers  —  and 
consistent  with  the  requirement  and  the  COEA.  And,  it  must  be  simple  to  allow 
specifics  to  be  moved  easily  between  documents. 

The  Air  Force  has  used  a  “Four  Star  Summit”  process  to  scrub  requirements 
separately  from  a  milestone  review.  The  focus  of  the  Summit  is  to  ensure 
requirements  are  clear,  that  the  system  meets  important  requirements  while  not 
spending  resources  at  the  point  of  diminishing  returns,  and  that  requirements  are 
operationally  testable.  It  was  beneficial  to  the  JointSTARS  program  by  focusing 
requirements  demand  and  allowing  consideration  without  the  DAB  process 
political  pressures.  Having  agonized  during  the  1991  Summit,  the  using  com¬ 
mand  in  1993  had  the  operational  rationale  for  every  critical  requirement  and 
could  respond  immediately  to  challenges  or  queries  without  wading  through 
historical  documentation  or  studies.  The  JROC  and  Army  and  Air  Force  chains 
of  command  supported  the  outcome,  making  it  clear  requirements  would  not  be 
reopened  for  the  DAB. 

IPS 

The  Integrated  Program  Summary  is  the  most  important  DAB  document,  cross- 
referencing  much  documentation,  and  touching  on  virtually  every  program 
aspect.  The  IPS  is  the  document  an  OSD  staffer  will  turn  to  first  in  writing  the 
Integrated  Program  Assessment.  The  program  office  should  spend  time  with  the 
guidance  in  DoD  5000.2  and  use  a  successful  example  before  starting  work. 

The  heart  of  Annex  A,  the  Program  Structure,  is  a  master  schedule,  which 
should  also  support  the  APB.  The  guidance  for  Annex  A  seems  to  suggest  it 
consists  of  one  chart,  but  as  staff  elements  engage  it,  it  may  grow.  Annex  B,  the 
Program  Life-Cycle  Cost  Estimate  Summary,  should  be  drawn  from  the  stand¬ 
alone  cost  documentation  submitted  to  the  CAIG,  as  specified  by  DoD  5000.2- 
M.  You  should  have  a  understanding  with  OSD  on  definitions  of  cost  elements, 
fiscal  years  to  be  shown,  base  years,  and  other  aspects  of  the  annex,  which  may 
change  at  the  last  minute  as  a  result  of  Service  or  OSD  Cost  Analysis  Improve¬ 
ment  Group  (CAIG)  reviews. 

Annex  C,  the  Acquisition  Strategy  Report  (ASR),  requires  a  substantial 
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Figure  4.  IPS  Cover  Sheet  Format 


amount  of  data,  including  broad  acquisition  strategy,  Federal  Acquisition 
Regulations  (FAR)  clauses  and  logistics  support  plans.  Though  interaction  with 
OSD  staff  will  vary  depending  on  the  reviewer,  the  Joint  STARS  lesson  learned 
is:  Follow  the  guidance  exactly,  and  be  prepared  to  provide  additional  informa¬ 
tion.  Policy  is  evolving,  and  there  are  differing  views  between  OSD  and  the 
services  on  who  determines  and  approves  the  acquisition  strategy,  vis-S-vis  the 
Acquisition  Plan  that  the  Service  Acquisition  Executive  (SAE)  approves.4 
Getting  to  the  DAB  on  time  suggests  flexibility  here  and  a  team  effort  between 
the  program  manager,  Procurement  Contracting  Officer  (PCO)  and  OSD 
reviewer. 

4  Change  1  to  DoDI  5000.2  specifies  ASR  approval  is  now  required  prior  to  release  of 
the  request  for  procurement  (RFP). 
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Annex  D,  the  Risk  Assessment,  is  another  up-front-and-early  document  It 
will  probably  be  widely  circulated  on  the  OSD  staff,  including  the  CAIG  for 
consideration  in  the  cost  estimate,  and  it  may  generate  questions  and  plant  visits. 
Annex  E,  Environmental,  is  somewhat  general,  and  environmental  concerns 
vary  with  current  events.  Get  your  program  office  or  product  center  focal  point 
together  with  the  appropriate  OSD  staffer  early,  and  stay  tuned  for  new 
legislation  and  regulations.  If  your  budget  does  not  show  compliance  with  a  new 
law,  it  is  not  OSD’s  problem! 

Though  DoD  5000.2-M  guidance  on  the  Affordability  Annex  seems  to 
require  charts  showing  your  program  in  relation  to  broad  aggregates,  expect 
direction  by  OSD  staff  on  what  comparisons  to  make.  The  OSD  analyst  will 
likely  be  more  interested  in  a  mission  area  rather  than  Service  topline.  As  the 
program  office  does  not  generally  have  access  to  mission  area  data,  you  will  need 
headquarters  help,  as  did  Joint  STARS.  Revising  the  annex  took  place  at  the 
eleventh  hour,  and  in  retrospect  soliciting  guidance  earlier  in  the  process  would 
have  been  worthwhile. 

Annex  G,  the  Cooperative  Opportunities  Document,  is  a  challenge  because  of 
the  political  nature  of  international  events  and  the  number  of  players.  For  Joint 
STARS,  no  one  would  offer  guidance  on  how  to  write  it,  but  there  was  no 
shortage  of  criticism  when  the  document  was  submitted. 

APB 

The  APB  is  the  only  one  of  the  Service-prepared  documents  the  Under  Secretary 
of  Defense  (Acquisition  &  Technology)  (USD(A&T))  signs.  It  consists  of  three 
sections  —  performance,  schedule,  and  cost  —  and  is  prepared  using  OSD 
software  that  controls  formats  and  the  change  process.  Since  it  is  virtually  the 
only  document  that  survives  the  DAB,  the  OSD  staff  will  review  it  with  a  fine 
toothed  comb. 

Check  the  guidance  to  ensure  the  minimum  milestone  set  is  covered,  but  also 
research  the  definitions.  Joint  STARS  revised  its  APB  to  include  “Required 
Assets  Available”  and  provided  what  seemed  to  be  an  appropriate  date.  Howev¬ 
er,  the  program  office’s  interpretation  did  not  conform  to  Air  Force  policy  and 
OSD  analysts  picked  up  on  it  when  the  milestones  appeared  out  of  sequence.  So, 
keep  your  definitions  simple,  and  ensure  milestones  are  consistent  with  all 
schedule  information.  Section  C,  cost,  will  likely  be  a  last-minute  update  due  to 
the  CAIG  process.  A  talking  paper  explaining  the  proposed  APB  will  help 
considerably.5 

TEMP 

The  TEMP  requires  the  cooperation  and  coordination  of  the  program  office, 


5  When  the  APB  is  updated  between  milestones,  it  must  be  accompanied  by  a  Baseline 
Change  Request  memo.  For  a  milestone  review,  it  is  just  another  DAB  document. 
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user,  test  activities  and  Services.  Joint  STARS  showed  that  meeting  OSD 
expectations  is  difficult,  and  that  DoDI  5000.2  requirements  amass  detailed 
annexes.  Even  with  strong  leadership  and  discipline,  the  TEMP  may  delay  the 
DAB. 

Joint  STARS  began  revising  the  draft  TEMP  before  the  DAB  clock  started, 
but  after  a  year  the  document  was  getting  dated.  The  draft  elicited  extensive 
comments  from  OSD  staff  reviewers,  including  some  offices  normally  not 
expected  to  see  the  document.  Though  an  acceptable  TEMP  was  submitted  in 
time  for  the  DAB,  it  required  an  OSD  meeting,  an  interagency  tiger  team  and  lots 
of  follow-up  to  get  the  signatures.  Lessons  learned  were: 

•  Assemble  an  interagency  team  empowered  to  make  decisions  for  their 
organization .  You  do  not  have  the  time  to  incorporate  action  officers’  input 
only  to  find  that  their  managements  have  different  views. 

•  Document  the  process  in  writing.  Task  people,  get  their  input  and  prepare 
signed  meeting  minutes.  Respond  to  each  point. 

•  Keep  the  program  director  and  Program  Executive  Officer  (PEO)  in¬ 
formed.  You  are  more  likely  to  need  general  officer  help  on  the  TEMP  than 
any  other  document. 

DT  Report 

The  DoDI  5000.2 contains  no  guidance  on  this  report,  except  to  say  that  it  reports 
the  results  of  DT.  Joint  STARS  prepared  such  a  report  a  year  before  the  DAB  for 
authorization  of  Advance  Procurement  and  had  developed  a  good  working 
relationship  with  the  OSD(T&E)  staff.  The  program  office  and  OSD  maintained 
an  open  line  of  communications. 

MER 

The  MER  presents  some  unique  challenges.  Its  preparation  is  a  using  command 
responsibility  requiring  support  from  the  program  office  and  every  affected 
command.6  All  players  and  independent  cost  teams  need  to  incorporate  data 
from  the  MER  into  the  Operations  and  Support  (O&S)  estimates.  Spend  some 
time  defining  ownership  and  the  chain  of  command.  Though  Joint  STARS 
avoided  show  stoppers,  a  kickoff  meeting  with  the  sponsor  and  manpower  shop 
followed  by  a  tasker  might  have  made  the  process  smoother. 


6  This  could  include  users  in  several  different  theaters,  any  depot  which  performs  item 
management  or  provides  government  furnished  equipment  (GFE),  Air  Training  Com¬ 
mand,  and  other  organizations. 
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OPERATIONAL  TEST  REPORT  OR  ASSESSMENT 
Although  the  operational  test  agency  reports  through  a  different  chain  of 
command,  the  program  office  has  a  direct  stake  in  the  report’s  content  and 
timeliness.  Even  with  the  arms-length  relationship,  open  communications  and 
cooperation  are  essential,  and  the  program  office  should  be  able  to  comment  on 
the  draft  before  it  goes  to  the  Pentagon.  Though  views  may  differ,  the  program 
office  can  help  eliminate  any  errors. 

MANAGING  THE  DOCUMENTATION  PROCESS 
Except  for  the  formal  DAB-minus-180  day  planning  meeting,  documentation 
preparation  will  consume  the  process  until  the  drafts  are  submitted  45  days 
before  the  DAB.  Preparing  documentation  is  worse  than  merely  time  consum¬ 
ing;  it  is  also  tedious.  If  all  drafts  are  submitted  on  time  and  satisfy  OSD,  chances 
are  the  DAB  will  occur  as  scheduled  or  close  to  it.  Otherwise,  it  will  be  delayed. 

Each  document  must  have  one  person  (editor)  responsible.  Each  must  be 
broken  down  into  components  so  that  each  author  understands  what  and  when 
to  feed  to  the  editor.  All  documents  must  march  to  the  same  schedule. 

Aside  from  scheduling,  the  major  issues  in  document  preparation  are  content 
and  consistency.  While  DoD  50001.2  provides  very  specific  guidance  for  some 
documents,  guidance  for  others  may  be  general  or  nonexistent. 

In  providing  a  foundation  for  the  document,  you  should  : 

•  Acquire  all  of  the  regulatory  references,  and  examples  from  other  programs 
that  have  recently  gone  through  the  process. 

•  Develop  a  vision  or  outline,  and  provide  it  to  the  team  members.  As  a 
document  editor,  you  are  likely  to  find  yourself  working  with  specialists 
who,  while  well  versed  in  their  technical  disciplines,  do  not  understand  the 
DAB  process,  the  purpose  of  the  document,  or  the  target  audience. 

•  Realize  quality  counts.  Several  Joint  STARS  documents  were  accepted 
with  no  change  between  draft  and  final  submission,  and  workload  for  the 
“final”  copies  was  limited  to  changing  the  cover  and  reproducing  the  body. 

•  Consider  establishing  a  management  information  system  strategy,  but 
make  sure  it  does  not  develop  into  a  choke  point. 

•  Contact  an  OSD  reviewer  for  guidance  and  perform  “expectations  manage¬ 
ment” 

Though  informal  contacts  can  help  the  process,  they  must  be  controlled 
through  a  program  of  fice  clearinghouse  that  understands  who  is  talking  to  whom 
and  ensures  only  reasonable  commitments  are  made.  Closing  the  loop  with  the 
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PROGRAM  OFFICE/SERVICE  DOCUMENTATION  SUMMARY 

Document 

Who  Prepares 

Guidance 

Who  Signs  or  Approves 

Mission  Needs  Statement 
(MNS) 

User 

DoD  5000..2-M 

Joint  Requirements  Oversight 
Council  (JROC) 

Operational  Requirements 
Document  (ORD) 

User 

DoD  5000.2-M 

Service  Chief  of  Designated 
Representative 

Integrated  Program 
Summary  (IPS) 

Program  Office 

DoD  5000.2-M 

PD,  PEO,  User,  SAE 

System  Threat  Assess¬ 
ment  Report  (STAR) 

Component 

Intelligence 

Command 

DoD  5000.2-M 

DIA 

Manpower  Estimate 

Report  (MER) 

User 

DoD  5000.2-M 
&  Supplemental 
Guidance  from 
OSD(FM&P) 

USD(A) 

Test  &  Evaluation  Master 
Plan  (TEMP) 

Program  Office 

DoD  5000.2-M 

PD,  PEO, OTA,  User,  SAE, 
DOT&E,  DDR&ECTE) 

Cost  &  Operational 
Effectiveness  Analysis 
(COEA) 

User 

DoD  5000.2-M 

Acquisition  Program  Base¬ 
line  (APB) 

Program  Office 

DoD  5000.2-M 

PD,  PEO,  SAE,  USD(A&T) 

Cost  Analysis  Require¬ 
ments  Document  (CARD) 

Program  Office 

DoDi  5000.4 
&  Supplemental 
Guidance  from 
OSDCAIG 

Program  Office  Life- 
Cycle  Cost  Estimates 

Program  Office 

DoD  5000.2-M 

SAE 

Development  Test  Report 

Program  Office 

No  Format 
Speeded 

Director,  Test  Activity 

Operational  Test  Report 

OTA 

No  Format 
Specified 

Commander,  OTA 

Table  1.  Program  Office/Service  Documentation  Summary 


Service  staff,  especially  the  PEO  action  officer  and  program  element  monitor, 
is  important  so  all  are  saying  the  same  thing.  For  the  IPS,  Joint  STARS  used 
briefing  charts  for  the  initial  outline  and  established  positions  on  sensitive  issues, 
helping  senior  management  assign  appropriate  resources  to  the  IPS  writing 
team. 

Joint  STARS  attempted  several  times  to  estabish  a  formal  review  process,  but 
this  added  no  value,  perhaps  because  an  impl  ied  certification  created  anxiety  and 
reviewers  were  reluctant  to  sign.  Staff  summary  packages  were  not  used  at  all, 
since  they  habitually  take  too  long  and  result  in  excessive  editing.  In  the  end. 
Joint  STARS  relied  heavily  on  the  program  office  DAB  coordinator  and  PEO 
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DOCUMENT:  INTEGRATED  LOGISTICS  SUPPORT  PLAN 

PURPOSE:  EXPLAIN  HOW  SYSTEM  WILL  BE  SUPPORTED 

GUIDANCE:  DoDI  5000.2  PART  7 
KNOWN  ISSUES: 


Figure  5.  Judicious  use  of  briefings  for  complex  or  high  risk  documents 
can  help  secure  guidance  and  resources  before  writing  begins 
and  also  expedite  final  review. 

action  officer  for  final  review  and  consistency  check,  and  this  informal  arrange¬ 
ment  worked  satisfactorily. 

The  documentation  schedule  must  include  a  window  for  early  OSD  review. 
This  lesson  was  relearned  the  hard  way  in  the  case  of  the  Acquisition  Strategy 
Report.  Through  an  intense  effort  by  the  program  office  and  PEO,  including  a 
two  day  turnaround  on  questions  and  a  Pentagon  meeting,  a  potential  delay  was 
avoided.  The  mechanics  of  sharing  draft  documentation  must  be  worked  through 
the  Service  headquarters,  as  some  documents  will  have  a  Service  stakeholder 
while  others  will  not. 

As  the  deadline  for  documentation  approaches,  the  Service  team  needs  to 
manage  last  minute  activities  carefull.  Just  understand  who  is  doing  what,  so  that 
the  day  after  the  deadline,  if  some  documents  are  missing,  at  least  someone 
knows  where  they  are! 

Managing  the  Schedule 

Managing  your  program  in  the  DAB  process  is  an  enormous  effort.  Because  of 
the  number  of  events  and  organizations  involved,  strict  schedule  discipline  is 
mandatory.  Using  a  schedule  (PERT  and/or  GANTT)  detailing  DAB  activities 
can  provide  management  assistance.  Activities  identified  would  include  re¬ 
quired  documentation,  briefings,  reviews,  tests,  and  any  other  others  which  must 
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be  completed  prior  to  the  DAB.  Each  activity  should  be  further  broken  into  a 
series  of  subtasks  and  measurable  events.  An  OPR  for  each  should  be  designated 
and  held  accountable. 

Working  with  external  organizations  is  particularly  critical,  since  they  have 
other  priorities  and  might  not  be  stakeholders  in  your  DAB  review.  Make  sure 
they  buy  in  to  your  schedule.  If  a  DAB-critical  milestone  is  in  jeopardy,  the 
external  organization’s  chain  of  command  may  be  able  to  make  a  difference,  but 
only  if  used  in  time. 

The  schedule  requires  daily  review  to  identify  which  OPRs  need  to  be 
contacted  to  discuss  progress  and  problems  being  encountered,  matters  crucial 
to  understanding  where  the  program  stands  towards  achieving  its  scheduled 
DAB  Review.  Keeping  up  with  the  schedule  is  a  full  time  job.  Finding  the  best 
person  to  do  it  is  the  most  important  assignment  a  program  manager  facing  a 
DAB  can  make.7  * 

ORCHESTRATING  DAB  EVENTS 
Preplanning  Meeting 

Perhaps  the  most  critical  event  is  the  one  not  shown  on  the  official  DAB 
schedule:  an  informal  preplanning  meeting  which  allows  the  OSD  and  Service 
teams  to  develop  preliminary  documentation.  Four  months  from  the  planning 
meeting  through  draft  document  submission  is  generally  not  enough  time  to 
prepare  a  document.  This  meeting  encourages  team  building,  and  the  Service 
team  begins  the  process  of  “expectations  management”  through  a  heads-up  on 
required  documentation.8 

Planning  Meeting 

As  specified  in  DoDI  5000.2,  the  planning  meeting  assesses  readiness  for  the 
DAB  and  plans  for  documentation.  The  program  manager  should  attend,  for  not 
doing  so  places  the  program  in  peril.  The  program  office  DAB  coordinator  can 
brief  the  program  overview  (including  why  the  program  is  ready  for  a  review) 
and  recommendations  for  what  documentation  should  be  required  and  the 
review  schedule.  (These  should  have  been  negotiated  earlier  with  OSD.) 

The  planning  meeting  output  will  be  a  memo  from  the  DAB  committee  chair 
to  the  DAB  Secretary  and  the  Service  Acquisition  Executive  (S  AE)  recommend¬ 
ing  the  program  go  forward,  establishing  the  required  documentation,  identify¬ 
ing  major  issues,  and  providing  specific  guidance.  For  example,  in  Joint  STARS 
program  concurrency  was  raised  as  one  of  several  issues,  and  the  Air  Force  was 


7  This  individual  was  a  support  contractor.  Other  program  officies  tend  to  appoint 
someone  at  the  Major  or  GS-13  level. 

8  The  Joint  STARS  preplanning  meeting  was  held  five  months  before  the  formal  plann  ing 
meeting. 
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requested  to  address  its  need  and  risks.  The  planning  meeting  does  not  schedule 
the  DAB,  but  it  does  indicate  that  a  DAB  may  be  forthcoming.  The  OSD’s 
official  posture  is  to  await  submission  of  suitable  draft  documentation  and  then 
update  the  DAB  calendar  after  the  documentation  review. 

The  4-1/2  half  months  between  the  planning  meeting  and  documentation 
reviewprovide  an  opportunity  to  work  issues  and  possibly  arrange  for  staff  visits 
to  the  program  office,  contractor,  or  both.  As  a  minimum,  you  can  count  on  visits 
by  the  independent  cost  team(s). 

Documentation  Review 

Upon  submission  of  draft  documentation,  OSD  will  schedule  a  documentation 
review.  The  program  manager  will  brief  to  the  outl  ine  specified  in  DoDI  5000.2. 
Joint  STARS  followed  the  outline  exactly.  Draft  documentation  submission 
kicks  off  an  intense  activity  period  for  the  program  and  OSD.  If  staffers  see  a 
substantial  amount  for  the  first  time,  or  if  it  does  not  comply  with  the  guidance 
memorandum,  you  risk  a  delay. 

The  documentation  review  outcome  will  be  a  memo,  and  you  can  expect  a 
number  of  action  items  and  off-line  activities.  Though  the  documentation  will 
prompt  many  comments  and  questions,  you  should  work  with  the  OSD  staff  to 
distinguish  between  these  and  specific  changes  that  need  to  be  made  to  the 
document  in  order  to  make  the  final  submission  acceptable.  Regardless  of  which 
category  the  comment  falls  in,  respond  specifically  and  document  the  answer. 

CAIG  Review 

TheCAIG  review  will  culminate  months  of  activities  by  independent  cost  teams. 
Because  of  congressional  criticism  regarding  the  “independence”  of  the  inde¬ 
pendent  cost  estimate  (ICE),  policy  and  procedures  are  changing  in  this  area. 
Salient  events  in  the  Joint  STARS  ICE  exercise  included: 

•  Preparation  of  a  draft  CARD  and  submission  at  the  formal  planning 
meeting, 

•  V  isits  by  the  Air  Force  ICE  team  from  the  Air  Force  Cost  Analysis  Agency 
to  the  program  office  and  the  contractor, 

•  Visits  by  the  OSD  CAIG  staff  to  the  program  office  and  the  contractor, 

•  Visits  by  both  agencies  to  the  proposed  depot,  as  well  as  to  an  AWACS 
operational  unit  for  analogies, 

•  Voluminous  amounts  of  paper  flowing  between  the  program  office, 
contractor,  and  ICE  teams  supplementing  the  CARD, 


174  -  Spring  1994 


Acquisition  Review  Quarterly 


Coining  Up  Golden: 

Defense  Acquisition  Board  Review  Guide  for  Program  Offices 


•  Submission  of  initial  cost  estimates  and  documentation  by  program  office 
and  Air  Force  ICE  team  by  the  financial  management  staff  in  the  Office  of 
the  Air  Force  Secretary  for  review, 

•  A  week-long  reconciliation  meeting  between  the  program  office  and  Air 
Force  ICE  team,9 

•  Revised  estimates  and  cost  documentation  by  both  parties  as  indicated 
during  reconciliation  discussions, 

•  Two  formal  Air  Force  CAIG  reviews,  with  an  additional  meeting  after  the 
first  CAIG  session  failed  to  resolve  differences  in  the  time  allowed, 

•  Adoption  of  a  Service  cost  position  by  the  Air  Force  CAIG, 

•  A  major  program  restructure  to  reduce  the  disconnects  to  a  level  which  the 
Air  Force  could  commit  to  fund, 

•  Submission  of  program  office  estimate,  Air  Force  ICE,  and  service  cost 
position  to  the  OSD  CAIG, 

•  OSD  CAIG  review, 

•  A  two  week  delay  to  the  CSC  review  to  allow  the  OSD  CAIG  to  review  the 
cost  documentation  on  the  restructured  program,  and 

•  Follow-up  meetings  after  the  OSD  CAIG  review,  the  Committee  review, 
and  the  DAB  to  establish  the  USD(A)  position. 

The  fundamental  process  issues  from  the  Joint  STARS  DAB  were  the 
relationship  between  the  OSD  ICE  and  the  Air  Force  ICE,  and  the  Service/OSD 
reconciliation  process.  On  the  former,  the  issue  revolved  around  whether  the 
OSD  team  actually  did  an  ICE,  vice  using  the  Air  Force  ICE  team  as  an  extension 
of  themselves.  On  the  latter,  the  process  simply  does  not  provide  for  a  Service/ 
OSD  reconciliation,  nor  is  it  clear  who  would  arbitrate. 

While  policy  makers  will  no  doubt  refine  the  process  in  the  next  few  years, 
the  program  office  must  treat  development  of  its  cost  estimate  and  support  to  the 
ICE  as  a  live-or-die  exercise.  Some  lessons  learned  or  relearned  were: 


9  Reconciliation  ensures  both  teams  costed  the  same  program  and  allocated  costs  to  the 
same  accounts.  It  is  not  arbitration. 
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ICE  PROCESS  AS  OF  APRIL  1993 


Figure  6.  ICE  Process  as  of  April  1993 


•  You  cannot  underestimate  the  resources  required  to  support  all  the  cost 
estimate  activities.  Augmentees  from  the  product  center’s  “home  office” 
can  make  a  significant  contribution. 

•  Teamwork  between  OSD,  the  Service  ICE  team,  the  Headquarters  staff,  the 
program  office,  and  the  contractor  are  essential. 

•  Though  the  CARD  will  never  be  sufficient  to  perform  an  estimate,  it  is  both 
necessary  and  useful. 

•  Strict  baseline  discipline  is  crucial,  especially  when  the  program  is  evolv¬ 
ing.  You  can’t  recover  in  time  if  teams  diverge  too  far  on  what  is  being 
costed. 

•  Your  cost  documentation  must  be  of  quality  to  allow  another  team  to 
reproduce  your  estimate. 

•  Since  the  ICE  will  without  fail  increase  your  required  line,  you  must  be 
prepared  to  adjust  available  funding  and  content  at  the  11th  hour  and 
support  your  new  program  estimate.  However,  be  aware  programs  can  be 
killed  due  to  cost  growth  which  is  really  unnecessary  risk  dollars. 
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•  Know  the  strengths  of  your  methodologies  and  stay  consistent  with  the 
storyline,  establishing  program  office  credibility  up  front. 

•  Be  prepared  for  in-depth  reviews,  to  negotiate,  and  to  take  action  items  - 
recheck  your  facts  off  line  rather  than  spontaneously  giving  an  incorrect 
reply. 

•  Where  you  cannot  agree,  at  least  agree  on  how  to  characterize  the 
differences. 

•  Use  a  quick  turnaround,  flexible  automated  costing  model  to  prepare  the 
estimate,  funding  profiles  and  documentation;  and 

•  Take  advantage  of  this  process  to  gain  insight  into  the  weaker,  less  defined 
areas  of  the  program  office  estimate  by  comparing  your  information  to  the 
independent  estimates. 

JROC  Review 

Since  there  were  no  issues  requiring  resolution,  a  JROC  review  did  not  take 
place.  For  those  programs  undergoing  a  JROC,  the  format  for  the  review  is 
canned,  and  you  and  the  using  command  should  obtain  a  current  template  from 
the  JROC  Secretariat. 

Air  Force  Systems  Acquisition  Review 
Council  (AFSARC)  Review 

The  Air  Force  meeting  equivalent  to  a  DAB  occured  the  week  before  final 
documentation  was  due.  It  approves  the  program  that  will  go  forward  to  OSD. 
Since  theprogram  was  already  providing  ACAT I  documentation,  the  AFSARC 
imposed  no  additional  unique  requirements.  However,  following  the  program 
office/Air  Force  CAIG reconciliation  theprogram  had  a$900M  disconnect.  The 
program  office  had  worked  out  a  rephasing  with  the  concurrence  of  Air  Combat 
Command,  and  it  was  the  rephased  program  (with  roughly  a  $300M  disconnect) 
which  the  AFSARC  approved  and  promised  to  funD,10  and  the  Program  Office 
Position  and  Service  Cost  Position  cost  position  thus  became  the  same. 

Final  Document  Submission  and  Committee  Review 

While  draft  documentation  is  defined  as  documentation  over  the  PEO’s  signa¬ 
ture,  final  documentation  for  the  IPS,  APB,  and  TEMP  must  have  an  SAE 


10  Strictly  speaking,  policy  requires  the  program  to  be  funded.  An  IOU  was  accepted 
because  the  ongoing  “Bottom  Up  Review”  made  it  impossible  for  the  Air  Force  to  offer 
offsets. 
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signature.11  The  OSD  staff  emphasized  the  need  to  submit  change-bar  versions 
from  the  drafts  for  some  of  the  major  documents,  and  complying  with  this 
request  was  worth  some  midnight  oil .  Accordingly,  the  program  office  provided 
change-bar  versions  and  a  talking  paper  summarizing  the  changes. 

Like  the  documentation  review,  DoDI  5000.2  also  specifies  the  format  of  the 
briefing  to  the  committee.  Several  days  before  the  review  the  PEO  and  commit¬ 
tee  chair  decided  the  briefing  should  focus  on  issues,  and  some  of  the  required 
topics  were  not  presented.  This  phase  of  the  review  process  is  where  teamwork 
between  OSD,  the  Air  Staff,  and  program  office  really  pays  off.  Though  OSD 
and  the  Air  Force  disagreed  on  a  number  of  issues,  the  program  director  knew 
what  they  were,  and  all  Air  Force  attendees  were  prebriefed  on  the  background 
and  Air  Force  position.  The  cooperation  between  the  PEO  and  Mission  Area 
Director  and  their  respective  staffs  was  outstanding.  Everyone  involved  had  only 
one  agenda:  a  successful  DAB.  During  this  period,  staffs  also  began  work  on 
finalizing  the  APB,  exit  criteria  for  the  next  phase,  and  the  wording  of  the 
Acquisition  Decision  Memorandum  (ADM). 

DAB  Review 

Though  DoDI  5000.2  suggests  a  briefing  outline  along  the  lines  of  the  IPS 
executive  summary,  the  Joint  STARS  DAB  briefing  was  issue  oriented.  The 
program  needs  a  “yes,”  and  unresolved  issues  risk  either  a  “no”  or  a  non¬ 
decision. 

Joint  STARS  had  an  unresolved  issue  of  program  cost.  Since  this  was  the  first 
DAB  for  the  newly  appointed  USD(A&T),  the  OSD  staff  had  no  feel  for 
expectations  or  even  who  was  invited.  Because  of  differences  in  cost  estimates, 
the  decision  was  delayed,  but  a  compromise  was  reached  a  few  days  after  the 
review  and  a  timely  ADM  was  received.  Completed  staff  work  also  allowed  the 
USD(A&T)  to  approve  exit  criteria  and  the  APB  in  the  ADM,  vice  assigning 
action  items  which  would  have  needlessly  extended  the  process  another  few 
months. 

SUMMARY 

Upon  assuming  command  of  the  Joint  STARS  program,  with  only  five  months 
to  go  before  the  DAB,  the  new  program  manager  called  everyone  into  the 
auditorium  and,  dressed  in  BDUs  and  helmet,  declared  war  on  the  DAB.  In  fact, 
prosecuting  a  DAB  is  probably  as  close  to  war  as  acquisition  people  can  expect 
to  get.  Like  war,  there  is  only  one  objective  and  the  stakes  are  high. 

In  the  case  of  the  Joint  STARS  program,  a  contract  for  advance  procurement 
(long  lead)  had  been  let  13  months  ago,  betting  that  the  DAB  would  not  only 


11  Only  certain  documents  require  signature  pages.  For  others,  signature  can  mean 
forwarded  with  a  cover  memo. 
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approve  production,  but  approve  it  in  time  to  avo  id  a  work  stoppage  and  possibly 
loss  of  funds  as  the  fiscal  year  end  approached. 

Like  war,  everyone  —  program  office,  contractor,  headquarters,  and  else¬ 
where  —  needs  to  execute  the  same  battle  plan.  If  your  program  is  sound,  if 
leadership  supports  you,  if  you  stay  focused,  and  if  you  keep  sprinting  until  the 
finish  line,  everything  will  work  out.  You  may  even  wind  up  with  a  better 
program.  Good  luck! 
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